robots exclusion protocolの例文

例文

1. As Monier pointed out, " You don't want the thing to index itself, " so Web sites can use something called the Robots Exclusion Protocol to tell web crawlers what to skip.
2. I tried using the http : / / www . archive . org Wayback machine on http : / / www . nationmaster . com / encyclopedia / Chuquet but nationmaster uses a robot exclusion protocol and is not indexed by Wayback . ( talk ) ] ] 09 : 33, 2 Sep 2004 ( UTC)
3. The "'robots exclusion standard "', also known as the "'robots exclusion protocol "'or simply "'robots . txt "', is a standard used by websites to communicate with web crawlers and other email harvesters, spambots, malware, and robots that scan for security vulnerabilities may even start with the portions of the website where they have been told to stay out.